Papers with unit tests
Reason-Code: Reliable Code Generation via Test-Driven Monte Carlo Tree Search (2026.acl-industry)
Copied to clipboard
| Challenge: | Large Language Models (LLMs) are widely used for code generation, but their performance degrades on complex tasks. |
| Approach: | They propose an inference-time framework that formulates code generation as a search process guided by execution feedback. |
| Outcome: | The proposed framework improves reliability without paying full cost of additional sampling under strict latency budgets. |
Unit Testing for Concepts in Neural Networks (2022.tacl-1)
Copied to clipboard
| Challenge: | Existing theories of language and cognition hold that these representations are structured in a compositional way and that the meanings of composite concepts (''gray car'') are inherited predictably from the meaning of the parts. |
| Approach: | They propose to test models for determining whether a system’s behavior is consistent with several key aspects of Fodor’s criteria. |
| Outcome: | The proposed models succeed on tests of groundedness, modularity, and reusability of concepts, but important questions about causality remain open. |
StepCoder: Improving Code Generation with Reinforcement Learning from Compiler Feedback (2024.acl-long)
Copied to clipboard
Shihan Dou, Yan Liu, Haoxiang Jia, Enyu Zhou, Limao Xiong, Junjie Shan, Caishuang Huang, Xiao Wang, Xiaoran Fan, Zhiheng Xi, Yuhao Zhou, Tao Ji, Rui Zheng, Qi Zhang, Tao Gui, Xuanjing Huang
| Challenge: | Existing work integrates reinforcement learning with compiler feedback to enhance code generation quality but the long code generated by LLMs makes RL exploration ineffective. |
| Approach: | They propose a framework that integrates reinforcement learning and compiler feedback to enhance code generation quality. |
| Outcome: | The proposed framework outperforms state-of-the-art approaches in corresponding benchmarks and integrates reinforcement learning with compiler feedback to improve code generation quality. |
Dynamic Scaling of Unit Tests for Code Reward Modeling (2025.acl-long)
Copied to clipboard
| Challenge: | Existing large language models struggle to produce accurate responses on the first attempt for complex reasoning tasks like code generation. |
| Approach: | They propose a lightweight yet effective unit test generator that scales unit tests based on problem difficulty. |
| Outcome: | The proposed approach significantly improves performance on three benchmarks. |
MATCH: Task-Driven Code Evaluation through Contrastive Learning (2025.findings-emnlp)
Copied to clipboard
| Challenge: | GitHub Copilot generates 46% of the code on GitHub. |
| Approach: | They propose a reference-free metric that uses Contrastive Learning to generate meaningful embeddings for code and natural language task descriptions. |
| Outcome: | This paper compares the performance of a new similarity score with existing metrics. |
SUT: Active Defects Probing for Transcompiler Models (2023.emnlp-main)
Copied to clipboard
Mengnan Qi, Yufan Huang, Maoquan Wang, Yongqiang Yao, Zihan Liu, Bin Gu, Colin Clement, Neel Sundaresan
| Challenge: | Existing datasets are often criticized for their lack of granularity, which can mask deficiencies in basic syntactic elements that humans care about. |
| Approach: | They propose a new program translation metrics that address basic syntax errors . they propose BLUE, CodeBLUE and computation accuracy metrics which address these errors based on a highly interpretable evaluation harness. |
| Outcome: | The proposed model passes the unit tests with a 26.15% pass rate compared to previous models . |
Beyond Outcome Verification: Verifiable Process Reward Models for Structured Reasoning (2026.findings-acl)
Copied to clipboard
| Challenge: | Recent work on reinforcement learning with verifiable rewards (RLVR) has shown that large language models can be substantially improved using outcome-level verification signals. |
| Approach: | They propose a framework where intermediate reasoning steps are checked by deterministic, rule-based verifiers. |
| Outcome: | The proposed framework achieves 20% higher F1 than state-of-the-art models and 6.5% higher than verifiable outcome rewards, with substantial gains in evidence grounding and logical coherence. |
SeDev: Structured Semantic Exploration for LLM-Driven Code Generation (2026.acl-long)
Copied to clipboard
| Challenge: | Large Language Models (LLMs) have shown remarkable capabilities in automating code generation, but they suffer from insufficient exploration of the vast solution space. |
| Approach: | They propose a large-scale LLM-driven code generation framework that efficiently finds high-quality solutions in only a few iterations. |
| Outcome: | The proposed framework outperforms baselines while maintaining reasonable time and computational costs. |